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NASA/ASEE MANAGEMENT SCIENCE FACULTY FELLOWSHIP PROJECT 
KSC MANAGEMENT TRAINING SYSTEM PROJECT 

TRAINING EVALUATION: FINAL REPORT 
Josd A. Sepulveda, Ph.D., P.E. 

EXECUTIVE SUMMARY 

In the area of management training, "evaluation" refers both to the specific evaluation instrument 
used to determine whether a training effort was considered effective, and to the procedures 
followed to evaluate specific training requests. 

This reports recommends to evaluate new training requests in the same way new procurement or 
new projects are evaluated. This includes examining training requests from the perspective of 
KSC goals and objectives, and determining expected ROI of proposed training program (does 
training result in improved productivity, through savings of time, improved outputs, and/or 
personnel reduction?). 

To determine whether a specific training course is effective, a statement of what constitutes 
"good performance" is required. The user (NOT the Training Branch) must define what is 
"required level of performance". This "model" will be the basis for the design and development of 
an objective, performance-based, training evaluation instrument. 

The process of determining training needs and asking from the users "What is the expected result 
of training? What may happen if no training is given?," is called a "Needs Assessment". Training 
satisfying the identified needs require a specific, objective, performance -based, training evaluation 
instruments for each training course. 

At KSC, a "Needs Assessment Focus Group" determined training needs and expected results of 
training. Their answers will help to design a performance-based training evaluation instrument to 
determine whether training is effective, i.e., whether it meets user’s expectations. The evaluations 
will be accomplished in terms of performance improvement, quality improvement, financial 
impact, reduction in defectives, team building accomplishments, and less employee turnover. 
Separate evaluations will be performed when the course ends and a few month later. 

A delayed evaluation will ask the supervisor: 

* Did training result in improved unit performance? 

* Did training result in improved individual performance? 

* Would you send somebody else to the same course? 

A separate delayed evaluation will ask the trainee: 

* Did your performance improve as a result of this course? 

* Should somebody else in your unit take the same course? 

Scanning equipment and form-processing software are recommended to mechanize the data 
gathering, analysis and reporting processes. 
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1.0 BACKGROUND 

A review of the existing procedures at KSC indicated that the evaluation of training 
courses is limited to KSC Form 13 (see Appendix). The results are not used for formal 
reporting. Major problems with this form are: 

• Link to KSC’s goals and objectives not clear. 

• No cost/benefits considered. 

• Not used to assess performance improvement. 

• Form is processed manually. 

Among other characteristics, the proposed Management Training System at KSC was 
envisioned to have the following evaluation-related properties: 

• Linked to KSC Mission, Goals and Objectives. 

• Has a performance and values-based focus. 

• Sets priorities based on expected results. 

• MeasureT results in terms of performance improvement, benefits, financial impact, 
quality improvement, team building accomplishments, and less employee turnover. 

• Tracks training and performance data. 


The evaluations of specific training courses will be accomplished in terms of performance 
improvement, quality improvement, financial impact, reduction in defectives, team 
building accomplishments, and less employee turnover. 

2.0 THE EVALUATION OF TRAINING 

In the area of management training, "evaluation" refers both to the specific evaluation 
instrument used to determine whether a training course, seminar, presentation, etc., was 
considered effective, (i.e, whether it met specific, performance-based, objectives); and to 
the procedures followed to evaluate new training requests. 

2.1 The Evaluation of New Training Requests 

A Training Branch’s objective is to help the Center to achieve all the benefits expected 
when personnel are prepared to operate in their environments to their full potential. As 
part of this objective, it is recommended that the Branch: 
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Participate in Training Needs Assessment and Priority setting 

• Perform Needs Analyses 

• Perform Method-Means Analyses 

• Supervise continuous evaluation and determination of training needs 

Evaluate training requests in the same way new procurement or new projects are 
evaluated 

• Examine training requests from the perspective of KSC goals and objectives. 

• Determine expected ROI of proposed training program. Does training result in 
unproved productivity (through savings of time, improved outputs, personnel 
reduction)? 

Determine if productivity improves as a result of investment in training. 

• Measure results (via a specific, performance-based, evaluation instrument) 

• Keep a continuous review of the training results and productivity gains 

2.2 The Financial Impact of Training 

Training is expensive, however, lack of training could be more expensive! There are a 
number of costs associated to a training effort: 

• direct: materials, instructor, tuition, cost for media, cost of employee travel 

W and per diem, employee salary during training, opportunity cost of 

foregone production 

• indirect: administrative, office space, computers, simulators, depreciation of 

facilities 

• intangibles: cost associated with potential failure of personnel to perform a task 

or job; savings associated to finishing a job early due to good 
performance by everybody involved 

3.0 PERFORMANCE-BASED TRAINING EVALUATION 

3.1 Training Effectiveness 

To determine whether training is effective, a statement of what constitutes "good 
performance" is required. This allows to determine if training produced enough of the 
desired result as to be considered "effective." 

The user (NOT the Training Department) must define what is "required level of 
performance". This "model" will be the basis for the design and development of an 
objective, performance-based, training evaluation instrument. 
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The process of determining training needs and asking from the users "What is the 
expected result of training? What may happen if no training is given?," is called a "Needs 
Assessment". Training satisfying the identified needs require a specific, objective, 
performance-based, training evaluation instruments for each training course. 

At KSC, a "Needs Assessment Focus Group" determined training needs and expected 
results of training. Their answers will help to design a performance-based training 
evaluation instrument to determine whether training is effective, i.e., whether it meets 
user’s expectations. The evaluations will be performed in terms of performance 
improvement, quality improvement, financial impact, reduction in defectives, team 
building accomplishments, and less employee turnover. Separate evaluations will be 
performed when the course ends and a few month later. 

3.2 The Evaluation Procedure 

It is not enough to ’pass’ a training course. Skill retention and improved performance 
over time are most important. Separate evaluations will be performed when the course 
ends and a few month later. 

The evaluation at course end will focus on the delivery and content of the course. It will 
also state the objectives and purpose of the training and ask whether those were 
addressed in the course. 

A delayed evaluation will ask the supervisor who originally approved the training request: 

• Considering the following objectives (... here goes what Focus Group establishes 
as expected results for training...), did this person’s performance improve as a 
result of this course? 

• Did training result in improved unit performance (through savings of time, 
improved outputs, quality improvement, reduction in defectives, team building 
accomplishments, less employee turnover, or personnel reduction)? 

• Would you send somebody else in your unit to take the same course? 

A separate delayed evaluation will ask the trainee: 

• Considering the following objectives (... here goes what Focus Group establishes 
as expected results for training...), did your performance improve as a result of 
this course? 

• Would you recommend that somebody else in your unit takes the same course? 
Thus, the proposed evaluation procedure will: 

• Examine training requests from the perspective of KSC goals and objectives. 
Determine if productivity improves as a result of investment in training. 

Keep a continuous review of the training results and productivity gains. 
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This will allow the Training Branch to document and report to management and to 
departments: 

* Time and resources needed to provide training 

* Employee time and Center resources spent in training 

* Evaluation results (aggregate values) 


4.0 EVALUATION FORM PROCESSING 

It is recommended that the form used to evaluate training courses be modified in such a 
way that, not only all the necessary information is collected, but that it is done in a 
mechanized way to speed up processing and reporting and to reduce data entry errors. lo 
do this, the following scanning equipment and software by National Computer Systems 
are recommended (prices and brochures are attached): 


OpScan Model 20 with Dual Ink Read 
Survey Network Software 
Freight/Installation 


$9,100 

$3,000 

$270 


Total 


$12,370 


The scanner is capable of processing 2,000 double-sided forms per hour. The Dual Ink 
Read head allows reading black and blue marks. The software permits the design and 
production (in a laser printer) of custom designed forms. 

If purchasing of these is not feasible due to budget constraints, UCF may be able to 
provide the service of producing and processing the evaluation forms. The results would 
be dumped in an ASCII file and the statistical analysis would be processed at KSC. 
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PARTICIPANT’S EVALUATION 



SUPERVISOR’S NAME 


PLEASE INDICATE YOUR EVALUATION OF THE SEMINAR IN TERMS OF THE FOLLOWING 


1. Overoll evoluotion of teminor. 

2. Sermnor content — omount of detail and extent of coverage. 


3. Level of presentation. 


4. Effectiveness of hand — out materials. 


5. Effectiveness of audio/visuol aids. 


6. lnstructor( s) knowledge of subject. 


7. Instructor s) effectiveness. 


8. Application to your job. 









